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(57) A portable head mounted display, HMD, (10) displays a virtual or augmented reality operating area e.g. 
for use in game playing. Two cameras (12, 14) placed on the HMD comprise a stereo vision depth sensing 
system to recognise and determine the distance to obstacles in the user*s field of view. Real obstacles in the 
display may be 'cloaked' by forming virtual object around them. The user can therefore move around the 
physical environment, using only the displayed reality as a guide without fear of colliding onto real objects. 
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VIRTUAL REALITY 

Field of the Invention 

This invention relates to virtual reality and, in particular, to creating virtual reality 
environments for use in virtual reality games and similar arrangements. 

Background to the Invention 

It is, of course, well known to provide computer game apparatus comprising a two- 
dimensional display screen on which is graphically defined game area which can be viewed 
by a player. The player can notionally move through the game area using controls such as 
those provided by a joystick, computer keyboard, cursor control device or the like. However, 
irrespective of how realistically the game area is portrayed on the screen, the fact that it is two- 
dimensional effectively prevents the player from feeling as if they are actually within the game 
area itself and, in any event, the screen/control interface involves player movements which are 
very different to the movements which are being simulated in the game, i.e., for example, the 
user presses a button to 'jump* within the game. 

In order to provide a more realistic game playing environment, virtual reality systems have 
been developed which generally comprise a head mounted display (HMD) through which a 
virtual world is three-dimensionally defined and can be viewed as such, giving the player a 
sensation of actually being present within the game area. The head mounted display typically 
has some form of motion sensor which causes the display to change as the player moves a part 
of their body, such as their head, thereby giving the user the sensation of looking around the 
virtual world which makes up the game area. 

Some virtual reality systems permit the player to move around a physical area, such movement 
being translated into movement within the virtual game area. For example, US Patent No. 
5 ,9 1 3 ,727 describes an interactive contact and simulation game in which a player and a three- 
dimensional computer-generated image interact in simulated physical contact. The apparatus 
includes control means for generating a simulated image of the player and displaying that 
image within the computer-generated game area, and a number of position sensing means and 



impact generating means are secured to various parts of the player's body. The player moves 
around a predetermined physical space, and the position sensing means determine the player's 
position accordingly, in response to which determination, the computer-generated image is 
moved accordingly, to give the player the sensation of actually moving through the game area. 
When simulated contact between the player's image and virtual objects in the game area is 
determined by the control means, the impact generating means positioned at the apparent point 
of contact on the player's body is activated to apply pressure to the player at that point, thereby 
simulating contact between the player and the virtual game area. 

One of the main disadvantages associated with such a system is that the player must be 
positioned in a relatively large, substantially empty physical space to prevent them from 
colliding with objects which they are moving around. In practice, this problem is generally 
overcome by providing some form of hand control which allows the user to simulate 
movement through the virtual reality game area by turning in the direction in which they wish 
to move and then using the hand control to give a 'move forward' command. 

The concept of "Z-keying" is known in the prior art as a technique for merging multiple 
graphics and video streams. The streams to be merged are assumed to come from coincident 
viewpoints. Each stream has a corresponding depth value. Pixel values for frames in the 
resulting merged stream are taken from the corresponding pixel location of the source stream 
with the lowest depth (z) value. As a result, virtual objects can be combined into real world 
scenes. 

Augmented reality systems also exist whereby the head mounted display is effectively see- 
through so that the user can see computer-generated graphical objects which are superimposed 
on physical objects present within the physical space in which the player is located, the 
computer-generated objects being substantially the same as the true physical objects in size, 
shape and nature. Such systems have the obvious advantage that the player can see the 
surrounding physical environment and is therefore free to move around without the fear of 
colliding with any objects which are present therein. However, one of the major disadvantages 



is of course that the true physical environment (say a living room or the like) and the objects 
within it are not as stimulating and exciting as a typical graphical games environment. 

We have now devised an arrangement which overcomes the problems outlined above. 
Summary of the Invention 

Thus, in accordance with the present invention, there is provided Interactive image display 
apparatus, comprising portable display means for displaying a virtual scene, means for 
generating an at least partially virtual scene including one or more virtual objects, and means 
for rendering said virtual scene in said display means for viewing by a user, wherein the 
apparent distance of any point in the user's field of view within the virtual scene to the nearest 
virtual object in said virtual scene is no further than the actual distance in the same direction 
of view to the nearest physical object within the physical space surrounding the user. 

In one embodiment of the present invention, the means for generating a virtual scene may 
comprise means for determining a depth map from the user's viewpoint of the physical space 
surrounding said user, and means for generating said virtual scene such that its virtual depth 
is no greater than the corresponding depth of said depth map of said physical space. The depth 
map preferably comprises a pixel image of the physical scene within the user's field of view, 
wherein each pixel represents the distance to the nearest physical object within said field of 
view. 

In contrast to the concept of "Z-keying" the apparatus of an exemplary embodiment of the 
present invention operates in a different manner in that it uses the determined depth 
information relating the surrounding physical world to cover at least parts of that physical 
world completely. In one embodiment, the apparatus operates to cover up the real world 
completely by explicitly generating a graphical world which is always closer than the real 
world equivalent. 



The apparatus beneficially comprises means for generating a new virtual scene in response to 
a change of the user's viewpoint, the new virtual scene preferably being displayed on display 
means as a smooth perturbation of the preceding rendered virtual scene from the preceding 
user viewpoint. 

A preferred embodiment of the present invention comprises a motion sensor for providing a 
sensory input of the relative change of position of the user, the sensed change of position being 
translated into a corresponding change in the virtual position of the viewpoint of the virtual 
scene, the motion sensor preferably being located on a user's head, when in use. 

In another embodiment of the invention, the means for generating a virtual scene comprises 
means for generating a (at least approximate) three-dimensional model of the physical space 
surrounding the user from the user's viewpoint, a three-dimensional virtual world being 
generated such that virtual objects appear to surround all of the physical objects within said 
physical space, and the virtual scene rendered in the display for view by the user provides a 
view of the said virtual world in accordance with the position of said user. The three 
dimensional model of the physical space may predetermined for a given physical space. 
Alternatively, the apparatus may comprise means for determining depth information from a 
user's viewpoint at any given time, said depth information relating to the physical space 
surrounding said user, and sensor means carried or worn by the user for determining the 
physical location and/or orientation of said user, said depth information and said information 
relating to the physical location and/or orientation of the user being used to generate a three- 
dimensional model of the physical space from the user's viewpoint. 

In any event, the apparatus preferably comprises sensor means for determining the user's 
position and orientation relative to the surrounding physical space, and/or imaging means for 
determining the user's position relative to the surrounding physical space. Such imaging 
means may be arranged to identify one or more markers carried or worn by a user so that their 
position can be determined accordingly, or such imaging means may comprise image 
capturing means carried or worn by the user, and include means for recognising one or more 
elements or locations within an image captured thereby, means for determining its relative 



location within a surrounding physical space and means for determining its relative location 
within a corresponding virtual scene by identifying the corresponding one or more elements 
or locations within said virtual scene. In any event, the imaging means may comprise stereo 
imaging means. 

In one exemplary embodiment of the present invention, a three dimensional model of the 
physical space surrounding the user can be updated to take into account changes in the 
surrounding physical space since said three dimensional model was generated, and, optionally, 
the first three dimensional model can be stored and retrieved for use when required. In this 
case, the three dimensional virtual model corresponding to said first three dimensional model 
of the surrounding physical space can beneficially also be stored and retrieved when required. 

In a first exemplary embodiment of the present invention, the virtual scene may comprise an 
entirely virtual environment which includes virtual obstacles or entities that appear to occupy 
at least the same space as the physical obstacles or entities within the surrounding physical 
space which a user would wish to avoid colliding with. 

In another exemplary embodiment of the present invention, the virtual scene may comprise 
an augmented scene, with only moving entities (such as people) within the physical space 
being represented within the displayed scene as virtual entities. In this case, the display means 
may be substantially opaque, the scene rendered in said display means for view by the user 
comprising an image of the surrounding physical space captured by image capturing means 
carried or worn by the user. Alternatively, the display means may be substantially transparent 
with opaque pixels being rendered in the display means in the form of a partial virtual scene 
for masking one or more areas of the surrounding physical space viewed by the user through 
the display means. 

In all cases, the display means preferably comprises a head mounted display (HMD). 



If required, the virtual obstacles or entities may be, or appear to be, larger than the physical 
obstacles or entities they are intended to represent, and optionally, additional virtual obstacles 
or entities may be provided within the displayed scene which do not represent obstacles or 
entities within the surrounding physical space. 

The apparatus preferably comprises means arranged to adapt said displayed scene, in real time, 
to the three-dimensional physical space in which a user is moving, and may include warning 
means, such as an audio or tactile signal, arranged to alert the user of any potential collision 
with a physical object or entity. 

The apparatus may include means (particularly where the apparatus comprises apparatus for 
playing games) for providing virtual moving entities within said displayed scene, said entities 
being perceived to move through the virtual environment but being constrained by the same 
physical environment as is the user and, in the case where there are two or more users 
operating within the same virtual and physical environments (and each having their own 
display means), each user is preferably represented in the virtual scene by a different virtual 
character, the position of which within the virtual scene is determined by the user's position 
within the surrounding physical space. In this case, the apparatus is preferably arranged such 
that the or each user can select to be represented within the operating area by one of a plurality 
of virtual characters, such as monsters, robots, animals, soldiers, etc., according to the nature 
of the game. 

The apparatus beneficially comprises means for replacing one virtual scene with another 
virtual scene which is adapted to the same physical layout of the physical space in which the 
apparatus is being operated. 

Further, the apparatus may comprise motion sensing means for sensing motion of a user within 
the surrounding physical space, means for magnifying said motion and means for translating 
such motion to motion within said virtual space such that motion of the user within said 
surrounding physical space appears as substantially fester motion within said virtual scene. 



It will be apparent that, unlike the situation with totally augmented reality systems, registration 
between real physical objects or entities and the virtual reality cloaking objects or entities 
intended to mark their presence and position need not be exact, nor is the speed of rendering 
the virtual reality cloaking objects or entities quite as critical as it is with augmented reality 
alignment. It will be appreciated that it is sufficient to ensure that the virtual reality cloaking 
objects or entities appear to present virtual surfaces which effectively surround the physical 
objects or entities with a large enough safety margin to prevent collision. In practice, it is 
really only necessary to ensure that the clear space perceived by the user in the virtual scene 
is in fact clear within the physical environment in which they are moving. This would at least 
give the user an opportunity to safely move through the physical environment by ensuring that 
all physical obstacles or entities are marked as such in the displayed scene. 

Of course, there is nothing to prevent the user from attempting to move through the virtual 
surface (thinking, perhaps, that it is simply a virtual object or entity which does not cloak a 
physical object or entity) and, therefore, colliding with a physical obstacle or entity in their 
path. For this reason, warning means, such as an audio or tactile signal, may be provided to 
alert the user of any potential collision with a physical object or entity. 

Many different types of virtual environment are envisaged. In the case where the apparatus 
forms part of a games system, the virtual scene could comprise, for example, one or more 
caves, tunnels or rooms in a building. The apparatus provides a high degree of realism in this 
case, because the user can move freely within their physical environment (without fear of 
collision with any obstacles), and such movement is translated into simulated movement 
through the virtual environment which is displayed in three dimensions. Any games might 
include the provision of virtual characters, such as monsters, enemies, etc. which are also 
perceived to move through the virtual environment but which are preferably constrained by 
the same physical environment as is the user. 

The present invention can, by its very nature, be used in a variety of different physical 
environments, including within a residential building, garden, garage, etc. In the case that the 
user only has use of a single confined space, such as a living room or bedroom, the apparatus 
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may comprise means for replacing one virtual scene with another virtual scene, as mentioned 
above, the second virtual scene being adapted to the same physical layout of the physical 
space in which the apparatus is being operated, i.e. both operating areas or virtual 
environments including virtual images which cloak the same physical environment layout. 

Brief Description of the Drawings 

An embodiment of the present invention will now be described by way of example only and 
with reference to the accompanying drawing which is a perspective view of a head mounted 
display system for use in apparatus according to an exemplary embodiment of the present 
invention. 

Detailed Description of the Invention 

In order to effectively 'cloak * obstacles present in the physical space surrounding the apparatus 
of the present invention with virtual reality images appearing in appropriate positions within 
the virtual area as viewed by a user during use, there are three main issues to be considered. 

Firstly, the three-dimensional free space around the user at any one time (in the direction in 
which they are nominally looking) needs to be mapped. Secondly, a three-dimensional virtual 
environment needs to be generated with its free space lying within the known free space of the 
surrounding physical space. Finally, any movement by the user within the physical 
environment needs to be monitored and fed back so that the virtual environment can be 
appropriately updated (while still conforming to the available free space within the 
surrounding physical environment). 

Referring now to Figure 1 of the drawings, a head mounted display 1 0 for use in an exemplary 
embodiment of the present invention comprises a depth sensor which can determine the 
definite free space in the field of view of the player. Depth sensors for mapping the profile 
of a predetermined environment or area, and therefore determining the definite free space 
therein (using, for example, sonar, laser range-finding or stereo vision camera systems) are 
well known. 
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In the head mounted display of Figurel , a stereo vision system is employed and, as such, two 
cameras 12, 14 are mounted on opposing sides of the front panel 16 of the head mounted 
display 10. The cameras 12, 14 capture images within their field of view, and a plurality of 
points on the images captured by both cameras 12, 1 4 are searched until two points are found 
(one from each image captured by a respective camera) which correspond to the same point 
on a three dimensional object appearing in the images. The position coordinates of these 
corresponding points on the images and the relative positions of the two cameras 12, 14 are 
used to provide data as to the specific point on the object in a three-dimensional space. By 
identifying a plurality of such corresponding points, the system can recognise the position, 
shape and size of a three dimensional object within its field of view. Using this method, all of 
the obstacles within the user's field of view can be recognised and the remaining free space 
can be determined. An operating area is generated having the same perceived free space as 
is available within the surrounding physical environment, any obstacles being marked or 
'cloaked' with virtual images which are in keeping with the theme of the displayed scene. 

A relative position sensor 18, comprising for example an accelerometer or gyroscope, is 
mounted generally centrally on the front panel 16 of the head mounted display 10. The 
relative position sensor measures relative motion of the user's head with respect to the 
evolving three-dimensional virtual environment. An (absolute) orientation sensor 20 is also 
mounted generally centrally on the front panel 16 of the head mounted display 10 to measure 
the orientation of the cameras 12, 14 with respect to gravity. 

The head mounted display 10 is retained on a user's head (so that the front panel 16 thereof 
covers the user's eyes) by means of a pair of side arms 22, similar to those used to retain 
conventional spectacles in position during use. 

Consider a simple exemplary embodiment of the present invention comprising a game based 
on the exploration of a plurality of tunnels. Within each of the tunnels is a number of rocks, 
and aliens (or the like) appear from behind the rocks. The player is required to shoot the aliens 
as they appear. 
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The stereo vision camera system can be used to determine the largest area of free space 
available (in any one view) and the orientation of its ground plane within the physical space 
surrounding the apparatus, and the apparatus then creates virtually the largest tunnel 
corresponding thereto and populates it with rocks and aliens. As the wearer moves their head 
and body, the relative motion is measured and the position of the previously-constructed 
tunnel is updated and evolved to agree with the updated depth map created by the stereo vision 
camera system according to the updated head position. As the user 'moves' through the 
tunnel, additional tunnel elements (such as branches, additional rocks, extensions, etc) can be 
added according to a simple, predefined tunnel modelling scheme and the evolving free space 
around the user. It should be noted that the evolution of tunnels need not be entirely 
geometrically consistent in order to give the impression of inhabiting a virtual world. 

In the case where there are two or more players, each wearing their own head mounted display 
10, each head mounted display 10 could be provided with an additional imaging system 
(similar to the stereo vision camera system described above and possibly sharing common 
elements therewith). Each player could be provided with an arrangement of markers (worn 
above the head, say) which are recognisable by the imaging system. Such an arrangement of 
markers would have a known geometry such that the imaging system could be used to 
determine the position and three-dimensional orientation of the markers (and therefore the 
associated player) with respect thereto (as is the case in automated motion capture systems). 
Using this information, the additional players could be 'cloaked* within the virtual 
environment by the generation of respective virtual images having the same approximate poses 
as the additional players. A similar marking and imaging system could be applied to cloak key 
pieces of equipment. For example, a stick used by a player could be portrayed in the virtual 
environment as a sword or similar weapon. 

In an alternative embodiment of the invention, the game could be played within an augmented 
world, with only the players being cloaked. In other words, instead of generating a virtual 
environment, the players effectively see the real world through, for example, a head mounted 
video and display system, and within this environment, real players are replaced in real time 
by cloaked counterparts using substantially the same techniques as are described above. 
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In the foregoing specification, the invention has been described with reference to specific 
exemplary embodiments thereof. It will, however, be apparent to a person skilled in the art 
that various modifications and changes may be made thereto without departing from the 
broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the 
specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense. 
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1. Interactive image display apparatus, comprising portable display means for 
displaying a virtual scene, means for generating an at least partially virtual scene 
including one or more virtual objects, and means for rendering said virtual scene in 
said display means for viewing by a user, wherein the apparent distance of any point 
in the user's field of view within the virtual scene to the nearest virtual object in said 
virtual scene is no further than the actual distance in the same direction of view to the 
nearest physical object within the physical space surrounding the user. 

2. Apparatus according to claim 1 , wherein said means for generating a virtual scene 
comprises means for determining a depth map from the user's viewpoint of the 
physical space surrounding said user, and means for generating said virtual scene 
such that its virtual depth is no greater than the corresponding depth of said depth 
map of said physical space. 

3. Apparatus according to claim 2, wherein said depth map comprises a pixel image of 
the physical scene within the user's field of view, wherein each pixel represents the 
distance to the nearest physical object within said field of view. 

4. Apparatus according to any one of claims 1 to 3, comprising means for generating 
a new virtual scene in response to a change of the user's viewpoint. 

5. Apparatus according to claim 4, wherein said new virtual scene is displayed on said 
display means as a smooth perturbation of the preceding rendered virtual scene from 
the preceding user viewpoint. 
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6. Apparatus according to claim 4 or claim 5 , comprising a motion sensor for providing 
a sensory input of the relative change of position of the user, the sensed change of 
position being translated into a corresponding change in the virtual position of the 
viewpoint of the virtual scene. 

7. Apparatus according to claim 6, wherein said motion sensor is located on a user's 
head, when in use. 

8. Apparatus according to claim 1, wherein said means for generating a virtual scene 
comprises means for generating a (at least approximate) three-dimensional model of 
the physical space surrounding the user from the user's viewpoint, a three- 
dimensional virtual world being generated such that virtual objects appear to 
surround all of the physical objects within said physical space, and the virtual scene 
rendered in the display for view by the user provides a view of the said virtual world 
in accordance with the position of said user. 

9. Apparatus according to claim 8, wherein said three dimensional model of said 
physical space is predetermined for a given physical space. 

1 0. Apparatus according to claim 8, comprising means for determining depth information 
from a user's viewpoint at any given time, said depth information relating to the 
physical space surrounding said user, and sensor means carried or worn by the user 
for determining the physical location and/or orientation of said user, said depth 
information and said information relating to the physical location and/or orientation 
of the user being used to generate a three-dimensional model of the physical space 
from the user's viewpoint. 

1 1 . Apparatus according to claim 8, comprising sensor means for determining the user's 

m 

position and orientation relative to the surrounding physical space. 
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12. Apparatus according to claim 8, comprising imaging means for determining the 
user's position relative to the surrounding physical space. 

1 3 . Apparatus according to claim 1 2, wherein said imaging means is arranged to identify 
one or more markers carried or worn by a user so that their position can be 
determined accordingly. 

14. Apparatus according to claim 12, wherein said imaging means comprises image 
capturing means carried or worn by the user, and includes means for recognising one 
or more elements or locations within an image captured thereby, means for 
determining its relative location within a surrounding physical space and means for 
determining its relative location within a corresponding virtual scene by identifying 
the corresponding one or more elements or locations within said virtual scene. 

15. Apparatus according to claim 12, wherein said imaging means comprises stereo 
imaging means. 

16. Apparatus accordingto any oneofclaims8to 13, wherein a three dimensional model 
of the physical space surrounding the user can be updated to take into account 
changes in the surrounding physical space since said three dimensional model was 
generated. 

17. Apparatus according to claim 14, wherein said first three dimensional model can be 
stored and retrieved for use when required. 

18. Apparatus according to claim 15, wherein the three dimensional virtual model 
corresponding to said first three dimensional model of the surrounding physical space 
can also be stored and retrieved when required. 
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19. Apparatus according to any one of the preceding claims, wherein said virtual scene 
comprises an entirely virtual environment which includes virtual obstacles or entities 
that appear to occupy at least the same space as the physical obstacles or entities 
within the surrounding physical space which a user would wish to avoid colliding 
with. 

20. Apparatus according to claim 1 , wherein said virtual scene comprises an augmented 
scene, with only moving entities (such as people) within the physical space being 
represented within the displayed scene as virtual entities 

2 1 . Apparatus according to claim 20, wherein said display means is substantially opaque, 
the scene rendered in said display means for view by the user comprising an image 
of the surrounding physical space captured by image capturing means carried or worn 
by the user, said apparatus including means for modifying said image after capture 
by replacing pixels in one or more areas thereof with their virtual image equivalents, 
prior to display. 

22. Apparatus according to claim 20, wherein said display means is substantially 
transparent with opaque pixels being rendered in the display means in the form of a 
partial virtual scene for masking one or more areas of the surrounding physical space 
viewed by the user through the display means. 

23 . Apparatus according to any one of the preceding claims, wherein said display means 
comprises a head mounted display (HMD). 

24. Apparatus according to anyone of the preceding claims, wherein the virtual obstacles 
or entities are, or appear to be, larger than the physical obstacles or entities they are 
intended to represent. 
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Apparatus according to any one of the preceding claims, wherein there are additional 
virtual obstacles or entities within said displayed operating area which do not 
represent obstacles or entities within the surrounding physical space. 

Apparatus according to any one of the preceding claims, arranged to adapt said 
displayed scene, in real time, to the three-dimensional physical space in which a user 
is moving. 

Apparatus according to any one of the preceding claims, including warning means, 
such as an audio or tactile signal, arranged to alert the user of any potential collision 
with a physical object or entity. 

Apparatus according to any one of the preceding claims, including means for 
providing virtual moving entities within said displayed scene, said entities being 
perceived to move through the virtual environment but being constrained by the same 
physical environment as is the user. 

Apparatus according to any one of the preceding claims, wherein, in the case where 
there are two or more users operating within the same virtual and physical 
environments (and each having their own display means), each user is represented 
in the virtual scene by a different virtual character, the position of which within the 
virtual scene is determined by the user's position within the surrounding physical 
space. 

Apparatus according to claim 29, arranged such that the or each user can select to be 
represented within the operating area by one of a plurality of virtual characters. 

Apparatus according to any one of the preceding claims comprising means for 
replacing one virtual scene with another virtual scene which is adapted to the same 
physical layout of the physical space in which the apparatus is being operated. 
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32. Apparatus according to any one of the preceding claims, comprising motion sensing 
means for sensing motion of a user within the surrounding physical space, means for 
magnifying said motion and means for translating such motion to motion within said 
virtual space such that motion of the user within said surrounding physical space 
appears as substantially faster motion within said virtual scene. 

33. Interactive image display apparatus substantially as herein described with reference 
to the accompanying drawing. 
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